Installing from the ESS Nexus repository with PiPy

ADCSng can be installed directly from PiPy without the need to download the source code, in order to do so, you need to configure the given PIP installation by editing your local Python Package Index configuration file. More information is available in [RD-PIP].

The file is accessible with: more ~/.pypirc what follows is the current configuration for ADCSng:

Installing the source code

The source code of ADCSng is under configuration control under Git. The BitBucket repository that hosts the code is the following: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/adcsng/browse, since this is a private repository access might not be granted, if so please contact ESS. Alternatively the source code can be obtained under request from spice@sciops.esa.int.

If the user is to install ADCSng from its source code it is recommended to use PiPy to do so (taking advantage of the Python setup file), in order to do so using the Terminal go to the top level adcsng directory, adcsng and type:

Please note that the --user parameter is optional; if you are using your user Python installation it needs to be indicated.

Once installed, you will be able to run ADCSng anywhere in your computer.

Source Code structure

A brief overview of the code structure for ADCSng is useful for its installation, configuration and usage:

User Configuration

As a user you only need to take care of the user configuration. The user configuration has to be edited and prepared by the user, it consists of a JSON file and an example from the test directory of ADCSng can be used as baseline (adcsng/adcsng/tests/bc/bepicolombo.json). Any name can be provided to the configuration file, the file extension has to be *.json. The configuration file will determine the directory structure that you need to setup for ADCSng. The recommendation is to create a directory outside of the ADCSng directory structure and to create one per intended ADCSng execution/project.

The following table describes all the available configuration items (please note that all the items are strings):

Once the configuration file has been generated it can be placed anywhere in your computer, it is recommended to put it in the same directory level as where you will define the directory structure of your ADCSng configuration. What follows is an example of an ADCSng configuration file:

If we consider the previous configuration file we could have a directory structure as follows:

Remember that you need to generate this directory structure for ADCSng will not generate it for you.

Developer configuration

The developer configuration should not be modified by the user.

• Grammar file: This fis a text file that specifies what files shall be placed in the Meta-Kernel. This files are expressed in a file naming parttern supporting (* and ?) in order to easily specifiy several files with same pattern. There are also some prefix keywords that we can use to specify the files to include in the MK.

  • date: Will add all the kernels to the MK where the source filename matches the given expression.
    Eg: date:MEX_SA_??????_V00??.BC

    Will add:
    '$KERNELS/ck/MEX_SA_201227_V0001.BC'
    '$KERNELS/ck/MEX_SA_201228_V0001.BC'
    '$KERNELS/ck/MEX_SA_201712_V0001.BC'
    '$KERNELS/ck/MEX_SA_201812_V0011.BC'
    ...

  • latesN: Will add the N latest kernels where the source filename matches the given expression.
    Eg:
    latest3:MEX_SA_??????_V00??.BC

    Will add:
    '$KERNELS/ck/MEX_SA_210104_V0001.BC'
    '$KERNELS/ck/MEX_SA_210107_V0001.BC'
    '$KERNELS/ck/MEX_SA_210109_V0001.BC'

  • exclude: Will exlude the specified file from the MK. No pattern is supported.
    Eg:
    exclude:em16_tgo_sc_smp_017_00_01_20190713_20190810_f20180215_v01.bc

  • yearly: Will add the latest CK kernel that matches the given pattern and only includes the latest of the year in the MK. Only supported with CK Kernels.
    
    

  • monthly: Will add the latest CK kernel that matches the given pattern and only includes the latest of the month in the MK. Only supported with CK Kernels.
    
    

  • weekly: Will add the latest CK kernel that matches the given pattern and only includes the latest of the week in the MK. Only supported with CK Kernels.
    Note: The weekly option can include kernels with coverage shorter that seven days for first week and last week of the year.
    
    

  • daily: Will add the latest CK kernel that matches the given pattern and only includes the latest of the day in the MK. Only supported with CK Kernels.
    
    

• adcsng_mission_config.json: This is the developer configuration file to specify how ADCSng behaves for each mission. Below some notes still TBD:

  • merge_sources: Merge sources in incoming into one kernel.

  • append_kernel: Append inputs in incoming to the latest kernel.

  • kernel_date_format: Specifies the date format to use in {start_date} and {end_date} of the kernel_schma. Suppoted formats: ""[YYYY|YY]MMDD[T][hh][mm][ss]" or "YYMM" or "YYYYMM" or "YYYY"

  • kernel_rotation: Creates a new CK kernel when merge_sources is True and the date in the source filename exceeds a time from last rotation or latest kernels of same type. To set the periodicity use values: Yearly, Monthly, Weekly, Daily. Only supported with CK Kernels. If kernel_rotation is specified, the appropiate (yearly:, monthly:, weekly:, daily:) prefix must be used instead of prefix "date:" to the CK pattern at grammar cofniguration, or unexpected MKs could be generated.
    
    Eg: Considering an input filename: MEX_SC_210101_SADE_M.TAB the configuration value "kernel_rotation": "Monthly", will merge and append to the latest kernel until the 1st every month, this days ADCSng will create a new kernel to continue the appending on it.
    
    Note: If append_kernel is set to True, the data of the source file will be appended to the previous Kernel until the first rotation is triggered, then no more appendings will be performed for this input type until a new execution of ADCSng is launch.
    Note: source_date_format configuration keyword it's requiered when using kernel_rotation.
    Note: The Weekly can generate kernels with coverage shorter that seven days for first week and last week of the year.
    Note: Take a look to key "kernel_date_format" to see if has enough resolution for that periodicity. Eg: "kernel_date_format": "YYYY" will not work for rotation periods smaller than a year.

  • kernel_rotation_since_date: Defines the date when the CK rotation becames active with the following format: '%Y-%b-%d %H:%M:%S.%f', ej: 2018-APR-14 00:03:19.636 . This date its only used to know if a CK was generated using the CK Rotation periodicity or is just a Ck generated before this rotation functionality. This is usefull to search the CK to be used for appending the TM data in case of the TM data covers a period before that the CK rotation was activated. For example, if we set this date to 1 January 2021 while having a Monthly rotation and the 2020 CK covers the full year, when a telemetry file of 1 September 2020 arrives, ADCSng cannot use the CK rotation to detect the previous kernel to use as append destination, because the existing 2020 file an the arrived CK will lie in different rotation periods. In this case ADCSng will append the Ck into the last CK which start date is before the "kernel_rotation_since_date" and also before the start date of the arrived CK.
    
    Note: In case of not defining this since date, ADCSng will not try to append TM data that needs to be appended to a CK generated before that when CK rotation was initially setted up.

  • source_date_format: Specifies the way to extract the date from the input/source file names. It can be defined as a string of a list of strings. In case of list all the formats will be tried. The first valid date parsed will be returned. To do this, each source_date_format string shall contain three parts separated by '|' with the following schema: start_index|length|format . Where start_index is the index of the first character of the date, the length is the length of the date, and format is the format to use to parse the date. You can read about available formats here: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior
    For filename: MEX_SC_201022_SADE_M.TAB , source_date_format => 7|6|%y%m%d
    For filenames: mpo_raw_hk_aocs_measured_attitude_20210811.tab and mpo_raw_hk_aocs_measured_attitude_highrate_20210810.tab => "source_date_format": ["34|8|%Y%m%d", "43|8|%Y%m%d"],

  • merge_kernel: TBC!! If True, concatenates the source files in a list of kernels files specified in confuguration key: source_list

  • extra_sources: List with file schemas to obtain the filenames to be copied to processed folder in that are linked to the TM file processed. Used for example to copy also de XML file for NOMAD LNO to processed folder.
    

  • tm_file: Defines the format of the telemetry files to allow ADCSng to digest them. It shall be filled with the following configuration:

    • delimiter: Delimiter to use in the TM ASCII files, Only supported blank spaces (" ") or commas (",")

    • time_format: Defines the time field format to be used. The TM file can contain several time refering columns, but only one it's going to be used as reference. Supported values are: "SCLK", "UTC", "TDB", "TIMESTAMP"

      • "time_format":"TIMESTAMP" : This time format will expect that the time is divided in 2 columns, one for seconds, and the other for microseconds. ADCSng will get the start date from the filename or the XML and will call it Tstart. Then gets the time of the first row and calls it T0, for subsecuent rows T1, T2 TN. The computed row time will be: (Tn - T0) + Tstart
        
        

    • header_lines: Number of file header lines to skip before starting the rows digestion.

    • source_date_format: The pattern to extract the start date from the source filename. Only for "time_forma t":" TIMESTAMP"

    • xml_date_node : If found, ADCSng will look for a XML file with same name that the source file. Later will use the node name and date format to extract a more precise start date for the source file.
      Eg: "xml_date_node":{ "xml_node":"start_date_time" , "date_format":"%Y-%m-%dT%H:%M:%S.%fZ" }, , there is a good example of this in the configuration of EM16 NOMAD LNO
      
      

    • data_factor: String or list with the data factors to evaluate and multiply by the row element to be processed. In case of starting with "=" the result of the evaluation will be directly assigned to the output, so you can create a sort of mappings using this method. The "element" variable contains the value of the element to be processed. The evaluation in done with Python Math.eval() method. In case of list, must have same number of elements that rot_axis.
      Eg: "data_factor":["=-67.14315404 if element==0 else -0.06000004", "=-0.92637659 if element==0 else -0.37000276"] , there is a good example of this in the configuration of EM16 NOMAD LNO

    • rot_axis: String or list with the number of the rotation axis index where place the result of evaluating the element and data_factor. In case of list, must have same number of elements that data_factor.
      Eg: "rot_axis":["2","1"] , there is a good example of this in the configuration of EM16 NOMAD LNO

    • data: Object with one keyword for each column in the TM file, supported keywords are:

      • utc: In case of defined, defines the column number for the UTC date time.

      • timestamp_seconds and timestamp_microseconds: Defines the columns numbers for seconds and microseconds. Only required when using "time_format":"TIMESTAMP".

      • scet: In case of defined, defines the column number for the SCLK time in string format (eg: "1/0630115375.18204") . Also is supported to specify a two integers separated by a comma (eg: "2,3") to specify the two columns to use to obtain the SCLK time in String format. In case of using two columns, the SCLK Kernel file will be processed to check that the SCLK only contains one partition and to obtain the delimiter for SCLK time String format.

      • qx, qy, qz, qs: In case of defined, defines the column to use for each quaternion element.

      • ang1, ang2, ang3: In case of defined, defines the column to use for each agular value. Not all of the three angles need to be use
        
        Eg:

        # The tm_file configuration:

        "tm_file":[

        {

        "delimiter":",",

        "time_format":"SCLK",

        "header_lines":"0",

        "data":[

        {

        "utc":"1",

        "scet":"2,3",

        "qx":"5",

        "qy":"6",

        "qz":"7",

        "qs":"4"

        }

        ]

        }

        ]

         

        # Will ingest the following TM data format:

        2021-01-09 00:00:02, 558230371, 34771, 0.3776260018, -0.5337548256, 0.7562258244, 0.0250367038, 0

        2021-01-09 00:00:10, 558230379, 34771, 0.3786481619, -0.5327883959, 0.7565528154, 0.0197456591, 0

        2021-01-09 00:00:18, 558230387, 34771, 0.3796652555, -0.5317918062, 0.7568635941, 0.0144616254, 0

        2021-01-09 00:00:26, 558230395, 34771, 0.3806717992, -0.5307625532, 0.757163167, 0.009169871, 0

        ....

More details on the developer configuration can be found on [RD-ANG].

Kernels and Output directories requirements

Regardless of the kernels/output directory structure you setup the following subdirectories for each path are mandatory in order to execute ADCSng:

• kernels: 'ck', 'spk', 'lsk', 'sclk', 'pck', 'ik', 'fk' and 'mk'

• output: 'ck', 'spk', 'sclk', 'mk'

If any of those kernels sub-directories is not present ADCSng will trigger an error and the execution will fail.

Processed directory requirements

The processed directory should have a given sub-directory structure. This structure is determined by the 'originators' of each input file. ADCSng will generate these sub-directories if they do not exist. The 'originator' acronyms can be found in the developer configuration. In general they are the following:

• man: Files coming from Mission Analysis (typically test trajectories)

• fdy: Files coming from Flight Dynamics (typically OEMs and AEMs)

• soc: Files coming from the Science Operations Center (typically planning AEMs)

• hkt: Files from the Housekeeping Telemetry typically processed by the SOC Downlink group

• tcp: Time Correlation Packets coming from the S/C

• pit: Files from the PI teams

Please note that the files under these directories are usually published in the ESA SPICE FTP. An example can be seen in the following link: ftp://spiftp.esac.esa.int/data/ANCDR/BEPICOLOMBO/

If the 'processed_dir' is not specified in the configuration file (is left empty) then the processed files will simply not be copied (this is the case for ExoMarsRSP for instance).